docs(backend): document backend support and workflows - #5929
Conversation
Coding-Agent: Codex Codex-Version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning-Effort: xhigh
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughDocumentation now covers seven backends, including TensorFlow 2 and PyTorch-Exportable, across backend selection, training, freezing, compression, model formats, inference detection, and Hessian handling. ChangesBackend documentation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@doc/train/training.md`:
- Line 23: Remove the shell prompt marker from each command-only example, or add
corresponding output, to eliminate MD014 warnings: update the TensorFlow 2,
PyTorch, and PyTorch-Exportable examples at doc/train/training.md lines 23, 30,
and 37, and the TensorFlow 2 and PyTorch-Exportable examples at
doc/freeze/freeze.md lines 21 and 52.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 16cb4b9c-60c4-416c-bbc6-ec382d110acd
📒 Files selected for processing (4)
doc/backend.mddoc/freeze/compress.mddoc/freeze/freeze.mddoc/train/training.md
There was a problem hiding this comment.
Pull request overview
This PR adds user-facing documentation for selecting and using the TensorFlow 2 (eager) and PyTorch-Exportable (pt-expt) backends, covering backend selection plus train, freeze, and compress CLI usage across the relevant docs pages.
Changes:
- Add TF2 and PyTorch-Exportable tabs to the training instructions.
- Add TF2 and PyTorch-Exportable tabs to the freeze and compress docs (including
--lower-kindmention for pt-expt). - Extend
doc/backend.mdto include the two backends, their model/checkpoint formats, and inference suffix detection notes.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| doc/backend.md | Adds TF2 and PyTorch-Exportable backend documentation (selection, file formats, detection). |
| doc/train/training.md | Adds TF2 and PyTorch-Exportable training command tabs. |
| doc/freeze/freeze.md | Adds TF2 and PyTorch-Exportable freeze command tabs and notes. |
| doc/freeze/compress.md | Updates supported-backends note and adds TF2 / PyTorch-Exportable compress tabs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Coding-Agent: Codex Codex-Version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning-Effort: xhigh
Document TF2, PyTorch-Exportable, and JAX availability across model and descriptor guides, and add DP/JAX compression routes. Coding-Agent: Codex Codex-Version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning-Effort: xhigh
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
doc/backend.md (1)
114-117: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftDocument the DPA4
.pt2ABI, not just the filename classification.
doc/model/dpa4.mddocumentsdp --pt freezefor DPA4 anddoc/backend.mdsays.pt2belongs to PyTorch-Exportable. The code docs show thesept.pt2artifacts are AOTInductor, but both ABIs can be classified as.pt2, while the PT and pt_expt code paths/ABI are distinct. Make the inference guidance explicit about “which.pt2” or rename/correct one contract to avoid loading the wrong graph type.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@doc/backend.md` around lines 114 - 117, Update the `.pt2` inference guidance in the backend documentation to distinguish DPA4 `dp --pt freeze` artifacts from PyTorch-Exportable/AOTInductor artifacts, explicitly identifying which ABI each `.pt2` file uses. Align the wording with the separate PT and pt_expt code paths so users do not load a `.pt2` graph through the wrong backend contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@doc/model/train-fitting-dos.md`:
- Around line 4-11: Update the DOS training documentation to cover TensorFlow 2
and PyTorch-Exportable routes alongside the existing dp --tf train and dp --pt
train examples, adding dp --tf2 train and dp --pt-expt train tabs. If either
backend cannot train DOS models, instead revise the supported-backends
description to explicitly mark it as conversion/inference-only, consistent with
JAX.
---
Outside diff comments:
In `@doc/backend.md`:
- Around line 114-117: Update the `.pt2` inference guidance in the backend
documentation to distinguish DPA4 `dp --pt freeze` artifacts from
PyTorch-Exportable/AOTInductor artifacts, explicitly identifying which ABI each
`.pt2` file uses. Align the wording with the separate PT and pt_expt code paths
so users do not load a `.pt2` graph through the wrong backend contract.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 2b1880b6-d0b4-460d-b54d-d31775212ee3
📒 Files selected for processing (24)
doc/backend.mddoc/freeze/compress.mddoc/freeze/freeze.mddoc/model/dpa2.mddoc/model/dpa3.mddoc/model/dpa4.mddoc/model/dprc.mddoc/model/linear.mddoc/model/overall.mddoc/model/pairtab.mddoc/model/train-energy-hessian.mddoc/model/train-energy-spin.mddoc/model/train-energy.mddoc/model/train-fitting-dos.mddoc/model/train-fitting-property.mddoc/model/train-fitting-tensor.mddoc/model/train-hybrid.mddoc/model/train-se-atten.mddoc/model/train-se-e2-a-tebd.mddoc/model/train-se-e2-a.mddoc/model/train-se-e2-r.mddoc/model/train-se-e3-tebd.mddoc/model/train-se-e3.mddoc/train/training.md
🚧 Files skipped from review as they are similar to previous changes (1)
- doc/freeze/compress.md
Document the distinct .pt2 lower-input ABIs and add the missing DOS training commands for TensorFlow 2 and PyTorch-Exportable. Coding-Agent: Codex Codex-Version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning-Effort: xhigh
Document the dense nlist exception used by PT-frozen deepspin DPA4 models. Coding-Agent: Codex Codex-Version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning-Effort: xhigh
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5929 +/- ##
==========================================
- Coverage 79.53% 79.28% -0.25%
==========================================
Files 1075 1075
Lines 126134 126134
Branches 4592 4598 +6
==========================================
- Hits 100315 100004 -311
- Misses 24164 24477 +313
+ Partials 1655 1653 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
wanghan-iapcm
left a comment
There was a problem hiding this comment.
This is a large and mostly accurate sweep -- I checked the great majority of the new support claims against backend registration at head and they hold up, including the ones that are easy to get wrong. Specifically confirmed correct: every dp --tf2 / dp --pt-expt / dp --jax flag against the aliases in deepmd/backend/*.py; the dp --pt_expt to dp --pt-expt and .xlo to .hlo corrections (both were real doc bugs); all the frozen-model suffixes against each backend's suffixes; dp --jax freeze --hessian; the JAX energy-loss-only and no---init-frz-model caveats; the TF2 and Paddle additions on dpa2/dpa3/se_e3_tebd; and the DPA4 note, where JAX really does have deepmd/jax/descriptor/dpa4.py but no jax/model/dpa4_model.py, so "descriptor-only" is exactly right.
Two claims do not hold. Both are inline.
One smaller thing I did not raise inline. The Hessian rewording in overall.md and train-energy-hessian.md is not wrong -- it names dp --pt freeze and the TorchScript route specifically -- but hessian_mode is honored only at deepmd/pt/model/model/__init__.py#L371-L374, and neither pt_expt/model/get_model.py nor dpmodel/model/model.py mention it, so PyTorch-Exportable cannot construct a Hessian model at all. Since this PR establishes that {{ pytorch_icon }} covers both, the page is now silent on pt_expt where it used to be blanket. Worth a clause if you touch these lines again; not worth a round trip on its own.
OutisLi
left a comment
There was a problem hiding this comment.
The current DPA4 native-spin multi-rank guidance contradicts both the implementation and itself. The graph-route/native-spin section of doc/model/dpa4.md says that native-spin .pt2 archives carry no with-comm artifact, have has_comm_artifact=false, and are single-rank only. At this head, _needs_with_comm_artifact explicitly admits a graph-lowered NativeSpinModelKind whose descriptor communicates across ranks; test_native_spin_needs_with_comm_on_the_graph_lower_only asserts that behavior; and DeepSpinPTExpt loads and dispatches forward_lower_with_comm.pt2 for multi-rank graph native spin. The same page also correctly states near the earlier multi-rank overview and the Limitations section that pt_expt native spin supports multi-rank. Please remove or update the obsolete single-rank paragraphs so this page documents one contract that matches the current export and C++ runtime.
OutisLi
left a comment
There was a problem hiding this comment.
The documented PyTorch-Exportable compression workflow does not actually put a compressed model into the executable artifact for the ordinary .pte example. Except for the special fused DPA1 graph .pt2 branch, enable_compression passes the original uncompressed model dictionary to deserialize_to_file and stores the compressed dictionary only as model_json_override. DeepEval reconstructs that model.json object for metadata and introspection, but energy inference runs exported_module or _pt2_runner, so the tabulated model is not executed. I reproduced the documented se_e2_a-style .pte workflow locally: the input and output exported graphs were identical (711 nodes each), the output executable contained no tabulate_fusion operator, while only model.json contained the compression state. Therefore model-compress.pte provides no compression speed or memory benefit even though this page and backend.md present pt_expt compression as generally supported. Please either export the genuinely compressed executable or document the narrow supported case instead of advertising the ordinary .pte workflow as compressed.
OutisLi
left a comment
There was a problem hiding this comment.
The newly documented JAX .jax frozen-model route is not usable by the normal DeePMD inference/test interface. Please correct the format contract or implement the missing loader support.
OutisLi
left a comment
There was a problem hiding this comment.
The newly documented TensorFlow 2 compression workflow fails for the normalized default se_e2_a configuration. The implementation or the documented support contract needs to be corrected.
wanghan-iapcm
left a comment
There was a problem hiding this comment.
Both points are addressed, and I checked the files at 3bc4ff06a rather than taking the replies at their word.
On the pairtab page: Paddle is gone from the title and from both notes, and DP is back on the linear-combination note and on linear.md. Since the fix introduced a claim I had not checked in the first round, I verified it too — the new "PyTorch-Exportable" on both lines holds, because deepmd/pt_expt/model/ carries both dp_zbl_model.py and dp_linear_model.py. The asymmetry between the two notes is also right rather than an oversight: JAX and TF2 appear on the short-range interpolation line, where each has pairtab_atomic_model.py and dp_zbl_model.py, but not on the linear-combination line, where neither has a dp_linear_model.py — only linear_atomic_model.py. And Paddle genuinely has none of them.
On the compression page: switching the headline example to -i frozen_model.jax -o compressed_model.jax removes the case that would have raised. The prose now points at .jax as the general lossless path and defers the StableHLO question to the descriptor pages; I confirmed that deferral is not a dead pointer, since train-se-e2-a.md and train-se-atten.md both state the type_one_side requirement for .hlo export. One small thing for the record only: your reply said the type_one_side requirement was documented for compressed .hlo export, and in compress.md itself it is referred out rather than stated. That is the better placement given the requirement is descriptor-specific, so I am not asking for a change — just noting the wording difference in case you expected it inline.
The Hessian point from my earlier review body is resolved as well. train-energy-hessian.md now says "Supported backends: PyTorch-TorchScript, JAX" and labels the tabs the same way, instead of the blanket PyTorch icon. That matches where hessian_mode is actually honored — deepmd/pt/, deepmd/jax/, and deepmd/dpmodel/model/make_hessian_model.py, with nothing on the pt_expt side — so the page no longer implies coverage that does not exist.
wanghan-iapcm
left a comment
There was a problem hiding this comment.
Both points are addressed, and I checked the files at 3bc4ff06a rather than taking the replies at their word.
On the pairtab page: Paddle is gone from the title and from both notes, and DP is back on the linear-combination note and on linear.md. Since the fix introduced a claim I had not checked in the first round, I verified it too — the new "PyTorch-Exportable" on both lines holds, because deepmd/pt_expt/model/ carries both dp_zbl_model.py and dp_linear_model.py. The asymmetry between the two notes is also right rather than an oversight: JAX and TF2 appear on the short-range interpolation line, where each has pairtab_atomic_model.py and dp_zbl_model.py, but not on the linear-combination line, where neither has a dp_linear_model.py — only linear_atomic_model.py. And Paddle genuinely has none of them.
On the compression page: switching the headline example to -i frozen_model.jax -o compressed_model.jax removes the case that would have raised. The prose now points at .jax as the general lossless path and defers the StableHLO question to the descriptor pages; I confirmed that deferral is not a dead pointer, since train-se-e2-a.md and train-se-atten.md both state the type_one_side requirement for .hlo export. One small thing for the record only: your reply said the type_one_side requirement was documented for compressed .hlo export, and in compress.md itself it is referred out rather than stated. That is the better placement given the requirement is descriptor-specific, so I am not asking for a change — just noting the wording difference in case you expected it inline.
The Hessian point from my earlier review body is resolved as well. train-energy-hessian.md now says "Supported backends: PyTorch-TorchScript, JAX" and labels the tabs the same way, instead of the blanket PyTorch icon. That matches where hessian_mode is actually honored — deepmd/pt/, deepmd/jax/, and deepmd/dpmodel/model/make_hessian_model.py, with nothing on the pt_expt side — so the page no longer implies coverage that does not exist.
Duplicate submission on my side; superseded by the identical approval below.
DPA4/SeZM is realized in the PyTorch backend but exports through the AOTInductor .pt2 path, not TorchScript. The backend rename to PyTorch-TorchScript was mechanically applied to this page earlier; restore accurate wording in the quick-start and freeze-to-.pt2 sections. Coding-Agent: opencode opencode-Version: 1.18.9 Model: ustc/deepseek-v4-flash Reasoning-Effort: max
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.
Suppressed comments (3)
doc/troubleshooting/howtoset_num_nodes.md:20
- This section attributes the torchrun-based distributed training workflow only to the PyTorch-TorchScript backend, but the PyTorch-Exportable backend also initializes distributed training when launched via torchrun/srun (pt_expt entrypoint). Consider wording that covers both backends to avoid implying
dp --pt-expt traincannot be launched with torchrun.
doc/train/parallel-training.md:98 - This section title and opening sentence scope DDP/torchrun parallel training to the PyTorch-TorchScript backend, but the PyTorch-Exportable backend also supports torchrun-launched distributed training. Updating the heading/intro to include both helps keep the backend documentation consistent.
## PyTorch-TorchScript Implementation {{ pytorch_icon }}
Currently, parallel training in pytorch version is implemented in the form of PyTorch Distributed Data Parallelism [DDP](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html).
DeePMD-kit will decide whether to launch the training in parallel (distributed) mode or in serial mode depending on your execution command.
doc/model/train-se-e2-r.md:79
type_one_side=falseis also unsupported in the PyTorch-Exportable backend forse_e2_r(pt_expt uses the dpmodelse_rimplementation, which raises NotImplementedError whentype_one_sideis false). The backend list here should include PyTorch-Exportable as well.
In the TensorFlow backend, {ref}`env_protection <model[standard]/descriptor[se_e2_r]/env_protection>` cannot be set to a non-zero value.
In the PyTorch-TorchScript, JAX, and DP backends, {ref}`type_one_side <model[standard]/descriptor[se_e2_r]/type_one_side>` cannot be set to `false`.
iProzd
left a comment
There was a problem hiding this comment.
Re-reviewed the current head after the DPA4/SeZM follow-up. The documentation now correctly separates the dp --pt training route from the AOTInductor .pt2 export/runtime path and explicitly states that DPA4/SeZM does not use ordinary TorchScript. The prior backend-contract concerns are resolved, and I found no new blocking issue.
OutisLi
left a comment
There was a problem hiding this comment.
One remaining backend-classification error is noted inline.
OutisLi
left a comment
There was a problem hiding this comment.
One additional backend-workflow omission is noted inline.
OutisLi
left a comment
There was a problem hiding this comment.
One additional PyTorch-Exportable training-workflow omission is noted inline.
OutisLi
left a comment
There was a problem hiding this comment.
One additional graph-freeze behavior mismatch is noted inline.
OutisLi
left a comment
There was a problem hiding this comment.
One additional backend-workflow documentation gap is noted inline.
OutisLi
left a comment
There was a problem hiding this comment.
One additional spin-scheme semantic mismatch is noted inline.
# Conflicts: # doc/model/train-se-atten.md
Reply and close the current review round on deepmodeling#5929: - backend.md: describe the DPA4/SeZM `dp --pt freeze` export as the separate AOTInductor route instead of a PyTorch-TorchScript route, so the overview agrees with dpa4.md. - change-bias.md: document the PyTorch-Exportable `dp --pt-expt change-bias` route for .pt/.pte/.pt2 inputs, with the multi-task limitation noted. - parallel-training.md: add the PyTorch-Exportable DDP subsection with a `torchrun ... dp --pt-expt train` example, separate from the PyTorch-TorchScript ZeRO/FSDP2 section. - dpa2.md: drop the unavailable dense (`nlist`, default) fallback advice and state the forced graph lower for graph-eligible DPA-2. - finetuning.md: document the TensorFlow 2 and PyTorch-Exportable `--finetune` workflows and their accepted formats/limitations. - train-energy-spin.md: scope the virtual-atom (sel-extension) guidance to the deepspin scheme, add the DPA4 native exception, and add DPA4 to the scheme-aware supported-descriptor list. Coding-Agent: opencode opencode-Version: 1.18.11 Model: ustc/deepseek-v4-flash Reasoning-Effort: max
|
Addressed the current round of review feedback in 708891e and merged latest master (resolving the merge conflict):
All review threads replied to inline and resolved. Validation: mdformat passed on all changed files, Coding agent: opencode |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 34 out of 34 changed files in this pull request and generated no new comments.
Suppressed comments (1)
doc/freeze/compress.md:122
- The DP compression note omits the
.ymlDPModel format.deepmd.dpmodel.utils.serialization.load_dp_model()explicitly supports both.yamland.yml, so the docs should list both to avoid implying.ymlinputs are unsupported.
DP compression accepts native `.dp` and `.yaml` models.
- compress.md: list `.yml` alongside `.dp`/`.yaml` for DP compression, matching load_dp_model's supported extensions. - train-se-e2-r.md: include PyTorch-Exportable in the backends for which `type_one_side=false` is unsupported (pt_expt `se_r` inherits the dpmodel implementation that raises NotImplementedError). - howtoset_num_nodes.md: note that torchrun-based multiprocessing training applies to both PyTorch-TorchScript and PyTorch-Exportable. Coding-Agent: opencode opencode-Version: 1.18.11 Model: ustc/deepseek-v4-flash Reasoning-Effort: max
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 34 out of 34 changed files in this pull request and generated no new comments.
Suppressed comments (3)
doc/train/finetuning.md:5
- The supported-backends note omits PyTorch-Exportable even though this page now documents a PyTorch-Exportable fine-tuning workflow below. This is misleading for readers scanning the page header; list both PyTorch variants here for consistency with the rest of the backend docs.
> **Supported backends**: TensorFlow {{ tensorflow_icon }}, TensorFlow 2
> {{ tensorflow_icon }}, PyTorch {{ pytorch_icon }}, Paddle {{ paddle_icon }}
doc/model/train-energy-hessian.md:118
- This warning states that PyTorch-Exportable cannot construct a Hessian model for freezing or inference, but the codebase includes explicit pt_expt Hessian-model support (e.g.
deepmd/pt_expt/model/ener_model.pyhasenable_hessian()and there are pt_expt Hessian tests). The doc should not make an absolute “cannot construct” claim here; at minimum, remove the pt-expt sentence or replace it with a precise, verified limitation about export/freeze behavior.
> The PyTorch-TorchScript freeze route does not preserve Hessian output. A
> PyTorch-TorchScript model frozen with `dp --pt freeze` is treated as a
> standard energy model. PyTorch-Exportable cannot construct a Hessian model
> for freezing or inference. The JAX backend can preserve Hessian output in a
> frozen model with `dp --jax freeze --hessian`.
doc/model/overall.md:64
- This warning claims PyTorch-Exportable cannot construct a Hessian model for freezing or inference, but pt_expt has Hessian-model support in code/tests (e.g.
EnergyModel.enable_hessian()andsource/tests/pt_expt/model/test_ener_hessian_model.py). The documentation should not state that pt-expt cannot construct Hessian models; remove or rephrase this sentence to match the implemented behavior.
> The PyTorch-TorchScript freeze route cannot output Hessians, and
> PyTorch-Exportable cannot construct a Hessian model for freezing or
> inference. The JAX backend can retain Hessian output in a frozen model with
> `dp --jax freeze --hessian`; see [Freeze a model](../freeze/freeze.md).
Summary
Closes #5919.
Closes #5920.
Closes #5923.
Supersedes #5930.
Validation
ruff format .ruff check .uvx pre-commit run mdformat --files ...for all changed Markdown filesdp --dp compress --helpdp --jax compress --helpgit diff --checkCoding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh
Summary by CodeRabbit